Multiple window, multiple mode image sensor

ABSTRACT

Systems, Methods, and Apparatuses for an image sensor and a control system to cooperate with each photodiode in individual pixels to allow multiple pixels in a set of pixels to operate in a different imaging-mode of operation simultaneously within multiple window regions of the image. The image sensor has multiple window regions each capable of operating in different operating modes. Each pixel contains multiple photodiodes. The imager is fabricated with an additional semiconductor layer containing one or more metallization layers for interconnections and providing active CMOS circuits for control.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC 121 and priority to, asa continuation in part of, U.S. application Ser. No. 16/175,662 (SRIRef. No. US-7276-3) titled “Extended dynamic range imaging sensor andoperating mode of the same,” filed on Oct. 30, 2018, which claimed thebenefit as a continuation application of U.S. patent application Ser.No. 15/238,063 (SRI Ref. No. US-7276-2) titled “Extended dynamic rangeimaging sensor and operating mode of the same,” filed on Aug. 16, 2016,which claimed the benefit of U.S. Provisional Patent Application Ser.No. 62/206,417 filed on Aug. 18, 2015 and entitled “Extended dynamicrange (XDR) CMOS pixel and operating mode under 35 USC 119.Additionally, this application claims the benefit under 35 USC 119 andpriority to U.S. provisional patent application Ser. 62/652,891, titled“Methods for enhanced imaging based on semantic processing and dynamicscene modeling,” filed: 4 Apr. 2018, all of which are incorporatedherein by reference in their entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH

This invention was made with government support. The government hascertain rights in the invention.

BACKGROUND OF THE INVENTION

The field of the invention relates generally to imaging devices. Moreparticularly, the present disclosure is directed to complementarymetal-oxide-semiconductor (“CMOS”)-based imaging sensors capable of awide range of operation.

Imagers typically include an array of pixels that containlight-sensitive elements commonly known as photodetectors, such as CMOSor charge-coupled device (“CCD”) sensors. In general, photodetectorsaccumulate charge in accordance with the incident light during what isknown as an integration period. If a photodetector becomes full, orsaturated, before the end of the integration period, additional lightfrom a bright scene striking the photodetector does not accumulate anyfurther charge, and stores no further information.

CMOS image sensors first came to the fore in relatively low-performanceapplications where scene dynamic range was low, and moderate to highnoise levels could be tolerated. A CMOS sensor technology enabling ahigher level of integration of an image array with associated processingcircuits would be beneficial to many digital applications such as, forexample, in cameras, scanners, machine vision systems, vehiclenavigation systems, video telephones, computer input devices,surveillance systems, star trackers, motion detection systems, imagestabilization systems, and high-definition television imaging devices.

SUMMARY

Systems, Methods, and Apparatuses for an image sensor and a controlsystem to cooperate with each photodiode in an individual pixel to allowmultiple pixels in a set of pixels to operate in a differentimaging-mode of operation simultaneously within each row making up theimage. The image sensor has multiple window regions. The set of pixelswithin a first row in a first window region can operate in a firstimaging-mode of operation while simultaneously the set of pixels withinthe first row in a second window region can operate in a secondimaging-mode of operation, and so on. Each pixel contains multiplephotodiodes. Each pixel is fabricated with an additional semiconductorlayer to generate and route wire traces for the control signals.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates an embodiment of an example circuit schematic of aCMOS pixel.

FIG. 2 illustrates an embodiment of a diagram of an example a CMOS pixelwith four photodiodes in a pixel on a substrate.

FIG. 3 illustrates an embodiment of a diagram of an example another CMOSpixel with a MIM gain capacitor.

FIG. 4 illustrates an embodiment of a diagram of an example a CMOS pixelin a substrate.

FIG. 5 illustrates an embodiment of a timing diagram of a mode of anoperation of a CMOS pixel.

FIG. 6 illustrates an embodiment of a block diagram of an example imagesensor having a set of pixels making up the image sensor to capture animage.

FIG. 7 illustrates an embodiment of a block diagram of a control systemand pixel array for an image sensor having a set of pixels making up theimage sensor to capture the image, where each pixel has an architecturethat includes multiple photodiodes forming the individual pixel similarto the example pixels shown in FIGS. 1-5.

FIG. 8 illustrates an embodiment of a block diagram of an example pixelin a base state.

FIG. 9 illustrates an embodiment of timing diagram of the control systemoperating a quad pixel reset (exposure control).

FIG. 10 illustrates an embodiment of a block diagram of a pixel in anunbinned imaging-mode of operation.

FIG. 11 illustrates an embodiment of a timing diagram of the controlsystem operating a single pixel reset.

FIG. 12 illustrates an embodiment of a block diagram of a timing diagramof the control system operating a single pixel read out.

FIG. 13 illustrates an embodiment of a block diagram of an example pixelcircuit in a binned imaging-mode of operation.

FIG. 14 illustrates an embodiment of a block diagram of a timing diagramof the control system operating four pixels binned together and readout.

FIG. 15 illustrates an embodiment of a block diagram of an example pixelcircuit in an extended dynamic range imaging-mode of operation.

FIG. 16 illustrates an embodiment of timing diagram of the controlsystem conducting an extended dynamic range exposure timing sequence forthe integration times for the four example photodiodes.

FIG. 17 illustrates an embodiment of a block diagram of a circuit makingup a pixel in a given imaging-mode of operation with a high gain.

FIG. 18 illustrates an embodiment of a block diagram of a pixel circuitin a given imaging-mode of operation with a low gain.

FIG. 19 illustrates an embodiment of a block diagram of an example pixelin a high frame rate imaging-mode of operation.

The foregoing and other aspects and advantages of the invention willappear from the following description. In the description, reference ismade to the accompanying drawings that form a part hereof, and in whichthere is shown by way of illustration preferred embodiments of theinvention. Such embodiments do not necessarily represent the full scopeof the invention, however, and reference is made therefore to the claimsand herein for interpreting the scope of the invention.

DETAILED DESCRIPTION OF THE INVENTION

FIGS. 1-5 discuss an example image sensor with a wide dynamic range andwill be discussed in more detail later.

FIG. 6 illustrates an embodiment of a block diagram of an example imagesensor having a set of pixels making up the image sensor to capture animage. The image sensor has an array 500 with columns and rows ofpixels. In this example, the image sensor, configured to capture animage, has five rows and columns of pixels (rows 1-5). Each pixel in agiven row can operate in a different imaging-mode of operation anddifferent timing than another pixel in that row of the image.

FIG. 8 illustrates an embodiment of a block diagram of an example pixelin a base state. The example pixel 100 has a quad pixel design with fourphotodiodes, PPD1-PPD 4, connecting to a common sense node. The pixel100 employs multiple photodiodes, e.g. PPD1-PPD 4, which each have anindependent transfer gate (transfer gate1-transfer gate4) to a commonsense node. The common sense node is used for reading out each of thefour independent photodiodes in this example. The common sense node canalso be used for combining charge from the two or more photodiodes toperform noiseless, charge domain binning. The pixel 100 employs a seriesmetal insulator metal (MIM) capacitor selectively connected to the sensenode after the reset operation. When the MIM capacitor is not connected,the sense node has a much lower capacitance and can achieve a highconversion gain to achieve lower readout noise. The sense node resetoccurs when both the RESET and the MIM gates are turned on. The signalis readout using the source follower and the row select transistor,producing a voltage signal on the column bus (which has a current sourceload). The four readout transistors are labeled: Source Follower, ROWSelect, MIM, and RESET.

FIG. 7 illustrates an embodiment of a block diagram of a control systemand pixel array for an image sensor having a set of pixels making up theimage sensor to capture the image, where each pixel has an architecturethat includes multiple photodiodes forming the individual pixel similarto the example pixels shown in FIGS. 1-5. The control system may includecomponents such as a controller having two or more decoders implementedin Analog/ADC circuits, a column and row control circuit for eachtransfer gate of the photodiodes, the MIM gain capacitor, the Resetswitch, and the Row select switch, column logic drivers, and a timingcircuit.

The control system can use two or more decoders. The two or moredecoders cooperate with a timer to direct control signals to control i)frame rate, ii) integration times of the multiple photodiodes, and iii)binning of the multiple photodiodes on a per pixel basis to allowmultiple pixels in the array 500 of pixels to operate in the differentimaging-modes of operation simultaneously in different windows, allwithin a time frame of one image frame.

The control system can use two or more decoders that are each configuredto control a different aspect of putting each pixel into itsimaging-mode of operation. For example, a first decoder cooperating withthe timer can direct the control signals to the transfer gates for eachof the multiple photodiodes and the gate for the Row select to tell theimage sensor what one or more pixel locations and what row of the imagesensor will be readout at that time. Also, a second decoder cooperatingwith the timer can direct the control signals to control an integrationtime for each photodiode and whether the gain capacitor should connector not connect to the common sense node to control gain.

The transfer gates, such as a transistor, metal oxide device, etc., foreach photodiode have control signals going to them to control theintegration time for that particular photodiode; and thus, how long thatphotodiode collects photons before its associated transfer gate allowsthe accumulated charge in the photodiode to be sensed at the commonsense node. Also, a control signal controls the gate connecting the MIMgain capacitor to the sense node. The MIM gain capacitor can beselectively connected to the sense node during a read out to modify thecapacitance of the sense node; and thus, have a lower gain (whenconnected) or higher gain (when not connected).

The column control signals can be coordinated with the row controlsignals to address and control the array of pixels. A column controlserial data bus can load logical 1 and 0 values as control signals toevery row, the transfer gates 1-4 of the photodiodes, and the MIM gaincapacitor, for each of the selected pixels. The selected pixels can beselected with a logical 1 control bit and/or default to a transfer gates1-4 to an off state and a high gain MIM state.

One or more additional metallization layers on the second semiconductorplane of the fabricated chip can be a control signal plane to generatecontrol signals and route wire traces/conductor paths. The controlsignal plane couples to logic and analog circuits capable of routing thedriver signals to the four transfer gates and the MIM switch. Thecontrol signal plane provides a spacious platform for the controlsignals to the transfer gates 1-4 and MIM capacitor signals from theimaging plane drivers. The control signal plane also provides the rowand column select signals. Each row time from the column control, can beloaded in coordination with the row control.

Referring back to FIG. 8, the imaging sensor captures lightingconditions in an image, selectively in any of its pixels, that extendover a wide dynamic range. Each pixel 100 includes two or morephotodiodes coupled to a sense node, each photodiode having connectedthereto a different transfer gate for selectively controlling a chargeaccumulation therein due to an incident light collected; a gaincapacitor, a Reset switch, and a Row select switch.

The image sensor and its pixels capture an image containing imagingconditions, such as i) low or bright lighting conditions that extendover a wide dynamic range, ii) objects requiring high frame rates orshort integration times to minimize motion blur, iii) etc. and thenpotentially selecting a different integration time for at least twophotodiodes in the pixel 100 based on imaging conditions captured by thepixel 100. Each photodiode is controlled using a different transfergate.

The readout circuitry connects to the sense node and includes a resettransistor, a source follower transistor, and a row select transistor.The readout circuitry connects to the sense node of the pixel 100. Acapacitance associated with the sense node can be modified during thereadout of the at least two photodiodes. The capacitance associated withthe sense node can be modified by selectively connecting a gaincapacitor to the sense node. The gain capacitor can selectively connectto the sense node and readout circuitry when measuring the chargeaccumulation for one or more of the photodiodes.

The control system is configured to analyze the scene being captured inan image. The control system selects different imaging-modes ofoperation for two or more pixels in a row based on imaging conditions inthe scene content, such as different light intensities and frame rateneeded, within the scene being captured by the plurality image pixels.The controller may operate the imaging sensor by:

selecting a different imaging-mode of operation for at least two pixelsbased on imaging conditions captured in the pixel 100;

performing a readout of the pixel 100, wherein a capacitance associatedwith the sense node can be modified during the readout of thephotodiodes; and

assembling an image using collected data signals reflective of theaccumulated charge and other imaging conditions obtained from aplurality of pixels from the readout.

MODES

Thus, the image sensor permits operating a CMOS image sensor withmultiple imaging-modes of operation simultaneously with different pixelson a given row of pixels in multiple windows in the image sensor. Thus,some pixels may run in extended dynamic range mode, some pixels may runin a high frame rate mode, etc.

The control system connects with each photodiode in the individual pixel100 to allow multiple pixels in the set of pixels to operate in adifferent imaging-modes of operation in different windows simultaneouslywithin each row making up the image. Note, the image sensor can havemultiple window regions making up a full field of vision of the imagecaptured by the image sensor. For example, the set of pixels within afirst row in a first window region can operate in a first imaging-modeof operation while simultaneously the set of pixels within the first rowin a second window region can operate in a second imaging-mode ofoperation, and so on.

The multiple windows, multiple mode image sensor permits operating aCMOS image sensor with multiple operating modes in multiple windowssimultaneously. Conventional row based readout CMOS image sensors maysupport windowing to introduce new operating modes but can only supportone imaging mode on a given row. This design overcomes this limitation.To achieve the desired multiple window, multiple mode operation, theimager is fabricated with at least a second semiconductor layercontaining one or more metallization layers for wire traceinterconnections and providing active CMOS circuits for control. Theimager uses image processing that optimizes multiple operating modeswithin each frame for improved imagery and image understanding.

The image sensor has a second semiconductor plane with one or moreadditional layers of metallization to route the wire traces/conductorpaths for the control signals to allow different modes of operation withthe example quad pixel/electrical circuit. The circuit architecture canbe the same for all of the pixels operating in different imaging modesof operation. The controller may control the timing of control signalsto each pixel 100 in each row of the imager in order to achieve theenvisioned imaging-modes of operation that include:

1.) An unbinned photodiode readout of photodiodes, each potentially witha different integration time for a high resolution.

2.) A binned photodiode readout of photodiodes with multiple photodiodeshaving their accumulated charge from their integration times binnedtogether for a high sensitivity.

3.) Subframe integration time control.

4.) A high frame rate for motion adaptive signal integration (MASI)and/or short integration time with high gain for motion blur reduction.

5) Extended dynamic range from low lighting conditions to bright sunnyconditions. And

6.) On a per readout basis of the photodiodes making up that pixel 100,use a high or low gain selection for any of i) the unbinned photodiodereadout mode, ii) the binned photodiode readout mode, iii) the highframe rate mode, and/or iv) the optimum extended dynamic range mode.

i. The unbinned photodiode readout mode can be an imaging-mode ofoperation using a readout of photodiodes from a pixel 100, where all ofthe photodiodes have a different integration time.

ii. The binned photodiode readout mode can be an imaging-mode ofoperation using a readout of photodiodes from a pixel 100 with multiplephotodiodes having their accumulated charge from their integration timesbinned together. Thus, binning may occur where the charge from multiplephotodiodes connecting to the common sense node is physically addedtogether, increasing the effective exposure, sensitivity; and thus, abetter resolution. The controller can bin any combination, for example,of two, three, or four photodiodes to collect and read out charge/signallevel. The controller and the timer cooperate to have transfer gates ofbinned photodiodes pulsed all at a same time with other binnedphotodiodes.

iii. A high frame rate for motion adaptive signal integration (MASI) canbe an imaging-mode of operation using a readout of photodiodes from apixel 100 with multiple photodiodes. The high frame rate MASI algorithmachieves a high dynamic range by combining high frame rate images ofvarying exposures. The high frame rate mode can still implement binning,unbinning, and extended dynamic range in a given pixel 100. Note, a highframe rate could also cut out a certain amount of rows or evenindividual pixels from being read to match a data throughput of theimage sensor. Thus, the first decoder can select a sub set of the rows,such as 60 rows, which will have their charge readout from a totalamount of rows, such as 80 rows, making up that image sensor.

iv. An extended dynamic range imaging-mode of operation can be animaging-mode of operation using a readout of photodiodes from a pixel100 with multiple photodiodes. The dynamic range is a metric of how wella sensor can measure an accurate signal at low light intensities all theway up until it reaches full well capacity.

A controller and a timer can enable a high or low gain selection, on aper readout basis, for i) the unbinned photodiode readout mode, ii) thebinned photodiode readout mode, iii) the high frame rate mode, and/oriv) extended dynamic range mode. Gain can be a setting that controls theamplification of the signal from the one or more photodiodes beingreadout. For high gain, the control signal to the gate for the MIMcapacitor is off during the readout. For low gain, the control signal tothe gate for the MIM capacitor is on during the readout at the sensenode.

FIG. 2 illustrates an embodiment of a diagram of a representation of aCMOS pixel 100 with four photodiodes in a pixel 100 on a substrate.

The four photodiodes are shown with their transfer gates (i.e. TG1). Thecommon sense node is labeled in the center (SN). The four readouttransistors are labeled: SF, ROW, MIM and RESET.

In an embodiment, each pixel 100 includes a substrate; two or morephotodiodes formed in the substrate that are coupled to a sense node.

Additional layers are added on the semiconductor layers for generatingand routing control signal wire traces on the fabricated chip. Theadditional layer(s) allows more room to fit enough control signals tocontrol each of the photodiodes, the gain capacitor, the gate for Resetand the gate for Row select in each pixel 100 on an individual pixelbasis.

The control system is configured to use one or more decoders cooperatingwith a timer and wire traces routed in a metallization layers making upthe image sensor. One or more of the pixels have at least four transfergates each coupled to a corresponding photodiode in that pixel 100. Theone or more decoders cooperating with the timer and the conductorsrouted in the silicon layer (different from the layer forming thephotodiodes) direct control signals to operate each pixel 100independently of another pixel 100, on a per pixel basis, in the row.

This second (or more) semiconductor planes can be added by waferstacking and wafer-to-wafer connections by means such as direct bondinterconnect technology.

The example first semiconductor layer can have the pixels (photodiodesand transfer gates and sense node and gain capacitor and sourcefollower, select, MiM and reset transistors). The example firstsemiconductor layer can be backside illuminated (BSI), with pixelcircuits on the frontside of the semiconductor. The second or moresemiconductor layers can have control circuits (timers) and readoutcircuits with metallization layers for routing. The two or moresemiconductor layers can be interconnected with direct bond interconnecttechnology (DBI) and through silicon vias (TSV).

As discussed, the control system uses one or more decoders cooperatingwith a timer to direct control signals to i) transfer gates for each ofthe multiple photodiodes in the individual pixel 100 to control anintegration time for that photodiode, ii) a gate for a gain capacitorconnecting to a common sense node to alter a gain applied to anaccumulated charge from one or more of the photodiodes being sensed andread out, and iii) a gate for row select. Thus, a pixel 100 such as afirst pixel, can be directed to operate in a first imaging-mode ofoperation in a first window and another pixel, such as a second pixel,in the same row can be directed via the control signals to operate in adifferent imaging-mode of operation in a second window.

To achieve the desired multiple window, multiple mode operation, anadditional plane is added to provide space for i) control signals from acontroller to the four transfer gates (transfer gate1, transfer gate2,transfer gate3, transfer gate4) and the MIM capacitor (shown, forexample, in FIG. 8) ii) for each pixel in the array iii) on anindividual basis.

In this quad pixel configuration, the transfer gates control the gatingof the charge signal to the common sense node. Control of these gatesand the row control signals (Reset and Select) permit the multipleimaging-modes of operation. These transfer gates control binning or notbinning, subframe integration control by transferring unwanted charge tothe sense node/reset drain, extended dynamic range by adjusting theeffective integration time of the four photodiodes, and readout at highspeed.

FIG. 9 illustrates an embodiment of timing diagram 900 of the controlsystem operating a quad pixel reset (exposure control).

The controller and the timer send control signals to activate the gatesfor the RESET switch, MIM capacitor, and transfer gates TG1-4 to all beon at the same time to reset the pixel. The decoder of the controllerworks with the timer to send a control signal to a row where the imagerwants to dump the charge in a photodiode to reset the integration timefor one or more photodiodes that have been accumulating charge up tothat point. In this example, charge is removed from all four of thephotodiodes and the Sense Node. The photodiodes are now in a base state,such as shown in FIG. 8, to start integration for any of the multiplephotodiodes in the pixel 100.

A decoder cooperating with the timer is now responsible for sub pixelintegration time (short, intermediate, or long) for each of thephotodiodes. The figures below will discuss various imaging-modes ofoperation and how each photodiode is controlled.

FIG. 10 illustrates an embodiment of a block diagram of a pixel in anunbinned imaging-mode of operation. In this example, each of the TGs forthe four photodiodes is pulsed all at different times from otherphotodiodes. In this example, the transfer gate 1 for photodiode 1 isreceiving a control signal of a logical 1 to have its charge to besensed at the sense node and is read out. Transfer gates 2-4 alsoreceive a control signal of a logical 0, so their correspondingphotodiodes are electrically cut off from the sense node at this pointin time and are accumulating charge. The control signal to the gate forthe MIM gain capacitor can connect or disconnect the gain capacitor tothe sense node on each photodiode readout in this pixel 100. Note, thecontrol signal for the gate for the Row select would also be a logical 1so that this pixel 100 in this row will have its charge for photodiode 1read out.

FIG. 11 illustrates an embodiment of a timing diagram 1100 of thecontrol system operating a single pixel reset. This is how thecontroller controls exposure. Both photodiode #1 and the Sense Node havetheir charge removed. In this example photodiode 1 is being reset tohave its charge dumped so it can then start accumulating charge when itsintegration control signal is sent such as shown in FIG. 10. Note, thegates for the RESET, MIM capacitor, and TG1 switches are all on at thesame time and the TG2, 3 and 4 are off. The end result is photodiode 1is ready to start of integration for Pixel #1.

FIG. 12 illustrates an embodiment of a block diagram of a timing diagram1200 of the control system operating a single pixel read out. In thisexample, the unbinned pixels, pixel 1 is being read out.

At a first step at time 1, the controller and timer send control signalsto turn on the gates of the RESET switch and MIM capacitor. Thecontroller and timer also send control signals to turn off the transfergates TG1, TG2, TG3, and TG4 of the photodiodes to off. Each photodiodenow accumulates charge. The controller and timer send control signals tohave charge removed from the Sense Node.

At a first step at time 1 and extending into time 2, the controller andtimer send control signals to clamp samples for the Sense Node RESET.

At a second step at time 3, the controller and timer send controlsignals to the transfer gate TG1 for photodiode 1 to switch to on. Thecontrol signals to the transfer gates TG2-TG4 for photodiodes 2-4 aremaintain to keep these transfer gates TG2-TG4 off. The control signalsto the RESET switch and gate for the MIM capacitor is maintained off.The accumulated charge for photodiode #1 is connected and sensed at theSense Node. That sensed charge is readout without the gain capacitoraltering the gain for that charge being read.

At a third step at time 4, the controller and timer send control signalsto sample the charge at the sense node for a readout.

FIG. 13 illustrates an embodiment of a block diagram of an example pixelcircuit in a binned imaging-mode of operation.

The control system uses the controller cooperating with a timer to sendcontrol signals to bin together any combination of two or morephotodiodes to collect and read out accumulated charge, at a same time,from the photodiodes binned together. Note, the controller cooperateswith the timer to pulse gates of all of the photodiodes binned togetherat a same time with its other binned photodiodes while a remainingamount of photodiodes in the individual pixel 100 have their controlsignals sent at a different time from the photodiodes binned together.Also in general, photodiodes diagonally aligned as fabricated in thepixel 100, e.g. PDs 1 & 3 or PDs 2 & 4 (see FIG. 2 and FIG. 4) in theindividual pixel 100 are binned together first. Then a third photodiodenot diagonally aligned could also be included in the binned photodiodesif three photodiodes where being binned together.

The combination of binned and unbinned photodiodes can be used in, forexample, an extended dynamic-range imaging-mode of operation. Thecontrol system uses the controller and the timer to cooperate to place apixel 100 in the row into the extended dynamic-range imaging-mode ofoperation while the controller and timer send control signals to anotherpixel in the same row to operate in a different imaging-mode ofoperation, such as an unbinned photodiodes or all photodiodes binnedmode of operation. When in an extended dynamic-range imaging-mode, thecontroller and the timer cooperate to send control signals to thephotodiodes in the pixel 100 such that at least two of the photodiodesare binned together. However, at least another photodiode in the pixel100 has a different integration time than the binned photodiodes in theindividual pixel 100. This way the photodiodes in the pixel 100 willcapture charge with different integration times and capture differentimaging conditions. The control circuitry can use the readout of thegiven photodiode or set of binned photodiodes that captures the bestimaging conditions for the content, such as bright light, low light,fast moving object, etc.

In addition, per readout of charge, from any of the binned photodiodesor unbinned photodiodes in the pixel 100, the controller has an optionof adding in or removing the gain capacitor from the sense node duringthat readout. Thus, the gate of the MIM gain capacitor can be pulsedeither On (logical 1) or Off (logical 0) depending on if gain is neededfor low lighting conditions captured by that photodiode being read out.

FIG. 14 illustrates an embodiment of a block diagram of a timing diagram1400 of the control system operating four pixels binned together andread out. In this example, the binned photodiodes, photodiodes 1-4 arebeing read together in one read cycle.

At a first step at time 1, the controller and timer send control signalsto turn on the gates of the RESET switch and MIM capacitor. Thecontroller and timer also send control signals to turn off the transfergates TG1, TG2, TG3, and TG4 of the photodiodes to off. Each photodiodenow accumulates charge. The controller and timer send control signals tohave charge removed from the Sense Node. Note, the controller and thetimer cooperate to start off the sequence by selecting and pulsing theRow switch slightly before pulsing any of the other components in thatpixel 100.

At a first step at time 1 and extending into time 2, the controller andtimer send control signals to clamp samples for the Sense Node RESET.

At a second step at time 3, the controller and timer send controlsignals to the transfer gates TG1-TG4 for binned photodiodes 1-4 toswitch to on. The control signals to the RESET switch and gate for theMIM capacitor is maintained off. The accumulated charge for photodiodes1-4 are all connected and sensed (binned) at the Sense Node. That sensedcombined charge is readout without the gain capacitor altering the gainfor that charge being read. The controller and the timer cooperate tobin all four photodiodes in this example to achieve the highestsensitivity.

At a third step at time 4, the controller and timer send control signalsto sample the charge at the sense node for a readout.

FIG. 15 illustrates an embodiment of a block diagram of an example pixelcircuit in an extended dynamic range imaging-mode of operation.

The controller and timer cooperate to place a pixel 100 into an extendeddynamic range imaging-mode of operation.

The multiple pixels, all having a same architecture, in the image sensorcan be used to implement a high dynamic range for this image sensor witha same set of pixels via use of binning of some photodiodes, unbinningof other photodiodes, controlling a length of integration time for thebinned and unbinned photodiodes, and optionally bringing in thecapacitance at the sense node.

In this example photodiodes 1 and 3 are binned together. The multiplepixels all have a same architecture, such as the example architecture inFIG. 15, but each pixel's internal photodiodes are configured to capturedifferent amounts of integration time and potentially add a capacitivegain or not, with each read out, in this single pixel 100 in a time ofone frame. The controller analyzing the scene can decide which readoutof the pixels during that frame achieves the optimal image quality forthat pixel 100.

Again, the controller and timer cooperate to allow control of each pixel100 and each pixel's photodiodes on an individual basis for each rowmaking up the image sensor. Thus, pixels even within a row of pixelsmaking up a window/region of the entire image sensor can operate in avery diverse manner through the addition of a layer used almostexclusively for generation and routing the control signal traces, andthen adding enough logic and circuits into the controller and timer toscale to handle controlling components in each pixel 100 on anindividual basis from other pixels neighboring that pixel 100. What thisdesign allows is controlling individual subframes within a pixel 100 andhaving multiple pixels operating simultaneously in different controlmodes, such as high frame rate, high dynamic range, etc. per row ofpixels to allow a full field of vision image for an entire image whilebeing able to control individual pixels within that full field of visionof the image captured by the image sensor to eliminate shadows by havingpixels recording low light conditions to operate in a binned mode, tominimize oversaturation of pixels recording bright light conditions tooperate in low gain mode, and pixels when recording moving objects tooperate in a high frame rate mode and/or short integration time toeliminate blur, while other pixels operate in an extended dynamic rangemode to capture the rest of the lighting and image conditions in thisfull field of vision of the image.

Each window region of the image can have an optimum image capturing viapixels in that region.

In an example, a pixel 100 may operate in a dynamic range mode in afirst window while a neighboring pixel operates in another mode such asa high frame rate in a second window. The control system uses thecontroller and the timer to cooperate to enable an extended dynamicrange imaging-mode of operation via i) having two or more pixels in therow operate in a binned imaging-mode of operation to capture charge fromthese pixels in the row that are in low light conditions in a firstwindow while ii) having one or more pixels in the row operate with ashort integration time and a gain capacitor connected to a sense nodefor a low gain to capture charge from these pixels in the image that arein bright light conditions without oversaturating in a second window, aswell as iii) having other pixels in that row in a third window operatein a high frame rate imaging-mode of operation to eliminate blur inthese pixels when recording moving objects in that pixel 100 to capturean entirety of the image.

FIG. 16 illustrates an embodiment of timing diagram 1600 of the controlsystem conducting an extended dynamic range exposure timing sequence forthe integration times for the four example photodiodes.

At time T0, the controller and timer coordinate to send a reset forphotodiodes 1 and 3 to dump their charge and start their integrationtime. The controller and timer coordinate the bin the timings forphotodiodes 1 and 3 to have a longest amount of integration time toacquire charge for a high sensitivity such as low light conditions.

At time TJ, the controller and timer coordinate to send a reset forphotodiode 2 to dump its charge and start its integration time. Thecontroller and timer coordinate for photodiode 2 to have an intermediateamount of integration time to be read out at the sense node.

At time TK, the controller and timer coordinate to send a reset forphotodiode 4 to dump their charge and start their integration time. Thecontroller and timer coordinate for photodiode 4 to have a shortestamount of integration time to connect the MIM capacitor to the sensenode in order to have low gain to acquire signal level withoutsaturation.

Combined, the operation of photodiodes 1-4 gives this pixel 100 anextended dynamic range of operation for many imaging conditions.

FIG. 17 illustrates an embodiment of a block diagram of a circuit makingup a pixel in a given imaging-mode of operation with a high gain. Thephotodiodes can be in any operating state. When the unbinned or binnedphotodiodes are being read at the sense node, then the controller andtimer can increase the gain on/amplify the measured accumulate chargebeing read out. The controller and timers send control signal to keepthe gate for the MIM capacitor off during a read out. Thus, when the MIMcapacitor is not connected, the sense node has a much lower capacitanceand can achieve high conversion gain to achieve lower readout noise.

FIG. 18 illustrates an embodiment of a block diagram of a pixel circuitin a given imaging-mode of operation with a low gain. Again, thephotodiodes can be in any operating state. When the unbinned or binnedphotodiodes are being read at the sense node, then the controller andtimer can pulse the gate for the gain capacitor to connect to the sensenode when the photodiode's accumulated charge is being read out. Thegain from this connection remains low.

FIG. 19 illustrates an embodiment of a block diagram of an example pixelin a high frame rate imaging-mode of operation.

A high frame rate for motion adaptive signal integration (MASI) can bean imaging-mode of operation using a readout of photodiodes from a pixel100 with multiple photodiodes. The high frame rate MASI algorithm,achieves a high dynamic range by combining high frame rate images ofvarying exposures. The high frame rate mode can still implement binning,unbinning, and extended dynamic range in a given pixel 100. Note, a highframe rate could also cut out a certain amount of rows or evenindividual pixels from being read to match a data throughput of theimage sensor.

In an example, the controller and timer cooperate to send controlsignals to enable a row, such as a first row, in the first window tooperate in an extended dynamic range imaging-mode of operation whileanother row in the second window operates in a high frame rateimaging-mode of operation based on sensed imaging conditions in thefirst row and imaging conditions in the second row.

As discussed, in some cases, possibly this entire row or just some ofthe pixels in this row are not selected to be read in order to increasea frame rate/readout of the rows of pixels making up an entire image.

If this row is selected to be read in an increased frame rate mode thenthe photodiodes can be binned or unbinned.

The control system can use two or more decoders cooperating with a timerto direct control signals to one or more rows in the window to operatein a high frame rate imaging-mode of operation while a first pixel inthe row operates with two or more photodiodes binned together, with asecond pixel in the row operates with all of its photodiodes unbinnedbased on sensed imaging conditions in the first pixel and imagingconditions in the second pixel.

Example Image Sensor with a Wide Dynamic Range

An imaging system for capturing light that extend over a wide dynamicrange and method for operating the same are provided. The imaging sensorincludes a plurality of pixels. Each pixel includes a substrate; two ormore photodiodes formed in the substrate that are coupled to a sensenode, each photodiode having connected thereto a different transfer gatefor selectively controlling a charge accumulation therein due to anincident light; and a gain capacitor.

In some aspects, the system and method include positioning a pixel toimage a scene described by light signals that extend over a wide dynamicrange, and selecting a different integration time for at least twophotodiodes in the pixel based on light signals received by the pixel.The photodiodes are coupled to a sense node, and each photodiode iscontrolled using a different transfer gate.

The method also includes performing a readout of the pixel using areadout circuit connected to the sense node. A capacitance associatedwith the sense node is modified during the readout of the at least twophotodiodes. The capacitance associated with the sense node can bemodified by selectively connecting a gain capacitor to the sense node.

The gain capacitor selectively connects to the sense node and readoutcircuitry for measuring the charge accumulation. The readout circuitryconnects to the sense node and includes a reset transistor, a sourcefollower transistor, and a row select transistor.

A controller is configured to operate the imaging sensor by:

selecting a different integration time for at least two photodiodes in apixel based on light signals received by the pixel;

performing a readout of the pixel, wherein a capacitance associated withthe sense node is modified during the readout of the at least twophotodiodes; and

assembling an image using signals obtained from the plurality of pixelsin the readout.

The two or more photodiodes may include p-type diodes. The size of eachpixel can be in a range approximately between 1 and 20 μm. The chargeaccumulated in two or more photodiodes of the pixel can be combined.

The gain capacitor may include a metal insulator metal (“MIM”)capacitor. The capacitance associated with the sense node can bemodified based upon a dynamically determined dynamic range of the scene.The capacitance is in a range approximately between 1 off/μm2 and 10fF/μm2.

The different integration times are selected for two or more photodiodesin the pixel to achieve the wide dynamic range for the pixel. The widedynamic range can be up to approximately 10,000,000:1. The integrationtime of one or more of the at least two photodiodes can be selectedbased on information from a previous image frame. The integration timecan be in a range approximately between 1 ms and 1 sec.

The advantages of CMOS imagers over CCD imagers are that CMOS imagershave a low voltage operation and low power consumption. Also, CMOSimagers are compatible with integrated on-chip electronics (controllogic and timing, image processing, and signal conditioning such as A/Dconversion). In addition, CMOS imagers allow random access to the imagedata. On-chip integration of electronics is particularly advantageousbecause of the potential to perform many signal conditioning functionsin the digital domain (versus analog signal processing) as well as toachieve a reduction in system size and cost.

Appreciable intra-scene light intensity variation poses a significanttechnical problem for current imaging technologies, resulting inoverexposed or underexposed regions that reduce image quality. Althoughsome approaches have aimed at addressing this issue by acquiringmultiple image frames with different exposure times, such multipleexposure approach can introduce artifacts due rapid changes in scenerybetween frames, thus reducing image resolution and quality. Otherapproaches have utilized pixel redundancy or introduced additionalcomponents that have undesirable cost and limited resolution andsensitivity.

Therefore, the present disclosure provides a novel imaging approachusing an imaging sensor capable of capturing images under variouslighting conditions. Specifically, using a multi-photodetector pixelconfiguration designed to achieve multiple integration times, high andlow gain, and charge binning, the herein provided imaging sensor canprovide a wide or extended dynamic range, particularly within singleimage frames. As may be appreciated, the present disclosure can providetechnological advancement for a broad range of applications, includingcommercial and military surveillance, medical imaging, machine vision,autonomic vehicle imaging, to name a few. In particular, the presentdisclosure can improve imaging for scenery that includes regions withhigh and low signal outputs.

In accordance with one embodiment of the present disclosure, a pixel 100having a possible Extended Dynamic Range is shown in FIG. 1. The pixel100 includes a substrate 102 and four pinned photodiodes 104 formed inthe substrate 102, wherein the photodiodes 104 are configured tointegrate charge generated due to incident light. As shown in FIG. 1,the light may be directed from the front, or top side of the pixel 100,as indicated by Δ, or the back, or bottom side, as indicated by λ′, orat any angles there between. Herein, the incident light A may includeambient or scenery lighting described by a panchromatic spectral range.In particular, the panchromatic spectral range may include wavelengthsin the visible portion, the near infra-red portion and ultravioletportion of the spectrum. As such, active photosensitive areas of thepinned photodiodes 104 may be responsive to the panchromatic spectralrange. In some embodiments, active areas of the pinned photodiodes 104may be responsive to a sub-band of the panchromatic spectral range, asdescribed in co-pending, commonly owned, U.S. patent application Ser.No. 14/796,265, filed Jul. 10, 2015, the disclosure of which isincorporated herein by reference in its entirety. To this end, the pixel100 may include a filter separating the sub-band (i.e. a color filter).Furthermore, the active areas of the pinned photodiodes 104 may beresponsive to a particular polarization. The pixel 100 may include afilter (i.e. a polarization filter) for polarizing the incident light.By way of example, the size of the pixel can be in a range approximatelybetween about 1 and about 20 μm, although other values may be possible.

Each of the pinned photodiodes 104 is connected to a different transfergate 106 in a configuration that allows independent control of chargeaccumulation for each of the pinned photodiodes 104. That is, differentintegration times may be dynamically selected for two or more of thepinned photodiodes 104 to achieve a wide dynamic range for the pixel100. In some aspects, integration times may be selected based oninformation from a previous image frame. By way of example, integrationtimes can be in a range approximately between 1 ms and 1 sec, with adynamic range of up to approximately 10,000,000:1, although other valuesmay be possible.

The pinned photodiodes 104 are coupled to a common sense node 108,allowing for noiseless, charge domain binning. That is, using a singlesense node 108 charge accumulated in each of the four pinned photodiodes104 may be selectively combined. Although FIG. 1 shows four pinnedphotodiodes 104 as an example, it may be readily appreciated that feweror more pinned photodiodes 104, each having a different transfer gate106, may be included in the pixel 100. Preferably, the pixel 100includes two or more photodiodes 104.

The pixel 100 also includes readout circuitry 110 connected to the sensenode 108 that is configured for measuring the charge accumulation in thepinned photodiodes 104. As shown, the readout circuitry 110 includes areset transistor 112, a source follower transistor 114 and a row selecttransistor 116. In addition, the readout circuitry 110 also includes aMIM transistor 118 (i.e., a MOSFET switch) and a gain capacitor 120. Oneterminal of the gain capacitor 120 is electrically connected to anintermediate node 122 between the reset transistor 112 and MIMtransistor 118.

By way of example, a layout of a pixel is illustrated in FIG. 2. Asshown, four pinned photodiodes (“PPDs”) are depicted in a quad-pixelconfiguration, each with their associated transfer gates (i.e. TG 1,TG2, TG3, and TG4). The sense node is labeled in the center (SN) of FIG.2. The four readout transistors labeled SF, ROW, MIM and RESET, refer tothe source follower, row, metal-insulator-metal and reset transistors,respectively. As shown, the size of each PPD can be approximately 4 μm×4μm.

Referring again to FIG. 1, the capacitance of the gain capacitor 120 canbe selected based upon a predetermined dynamic range of the imagingsensor. For example, the capacitance may be in a range approximatelybetween 1 fF/μ^(m2) and 10 fF/μ^(m2), although other values may bepossible. In some implementations, the gain capacitor 120 may be ametal-insulator-metal capacitor, formed by a multi-level metal stackwith dielectric thickness configured to achieve high capacitance perunit area as described in co-pending, commonly owned, U.S. patentapplication Ser. No. 13/169,242, filed Jun. 27, 2011, the disclosure ofwhich is incorporated herein by reference in its entirety. By way ofexample, a layout of the quad-pixel configuration showing ametal-insulator-metal (“MIM”) gain capacitor is illustrated in FIG. 3.

During operation of the pixel 100, the MIM transistor 118 may be used toselect a high or low V/e− gain by selectively allowing the gaincapacitor 120 to be connected to the sense node 118. In this manner, awide dynamic range may be achieved. Specifically, when the gaincapacitor 120 is connected to the sense node 108 after a resetoperation, a low gain, higher saturation level can be achieved for thepixel 100. If the gain capacitor 120 is not connected, the sense node108 has a much lower capacitance that allows a high conversion gain fora lower readout noise. The sense node 108 reset occurs when both resettransistor 112 and the MIM transistor 118 are activated. In someembodiments, the pixel 100 may also include capabilities for addressingblooming effects, which occur when charge capacity is exceeded andadditional charges spill into neighboring pixels. For example, the pixel100 may include a clamping circuit (not shown in FIG. 1), as well asother circuits or capabilities.

As shown in FIG. 1, the pixel 100 may be configured as a p-type CMOSpixel. That is, the pinned photodiodes 104 are based on p-typesemiconductor materials. However, in another embodiment, theconductivity type of the pinned photodiodes 104, and other doped regionsof the pixel 100 may be reversed to form a functioning n-type CMOSpixel. In an n-type pixel (otherwise known as an NMOS pixel), thecarrier type is electrons, while in a p-type pixel (otherwise known as aPMOS pixel), the carrier type is holes. In some embodiments, the n-typeor p-type pixels/imagers may be either front-illuminated orback-illuminated.

In some embodiments, an imaging system for imaging a scene described bylight signals that extend over a wide dynamic range is provided. Theimaging system may include an imaging sensor or imager that is formedusing one or more arrays of pixels, as described with reference toFIG. 1. The imaging system may also include a controller for controllingthe operation of the pixels in the imaging system. In particular, thecontroller may be configured to read out signals from each pixel usingits associated source follower, reset and row select transistor, asdescribed, each pixel producing a voltage signal on a column bus (whichhas a current source load). Readout of the pixels in the imaging sensormay be performed using a variety of techniques known in the art. Forexample, a rolling shutter integration technique may be performed by thecontroller. The controller may then be configured to assemble one ormore image frames using the signals from multiple pixels in the imagingsensor.

In some aspects, the controller may be configured to analyze a scene,and based on scene content, such as regions of different lightintensities, adapt the operation of different pixels in the imagingsensor for optimal sensitivity. For example, the controller may modifydurations of exposure, integration times, and gain settings ofindividual pixels based on the light intensity information. In oneimplementation, the controller may be configured to select a differentintegration time for different photodiodes in a pixel based on lightsignals received by the pixel, and perform a readout of the pixel, wherethe capacitance associated with the sense node is modified during thereadout to modify the gain. In particular, the capacitance may bemodified based upon a predetermined dynamic range of the scene, or basedon analyzed scene content. In one example, the controller may operate aquad-type pixel using 4 different integration times, read out from theshortest integration time to the longest, and at least one of theshorter times being operated with the capacitance, i.e. gain, beingmodified at least once. In another example, the controller may operate aquad-type pixel using 3 different integration times, two pixels havingthe same integration time and signals therefrom being binned together,with the gain being modified at least once during the readout. These aremere examples, and it may be readily appreciated that any combination ofoperational modes are possible.

In some aspects, different pixels in an imaging sensor may be responsiveto different sub-bands of the panchromatic spectral range, as described.Therefore, the controller may be configured to operate respective pixelsin the imaging sensor in accordance with the different sub-bands.

Referring now specifically to FIG. 4, an image illustrating a mode ofoperation of a pixel 400 in a quad-pixel configuration, in accordancewith aspects of the present disclosure, in shown. In particular, toachieve a wide dynamic range for single image frames, as described, thepixel 400 may be operated using three signal readouts, each having adifferent integration time, although other signal readouts may bepossible. For instance, to capture the low-range imagery photodetectors“1” and “4”, located on the left 402 of the top row 406 of the pixel 400shown in FIG. 4, may be operated at the 4 fps integration time of 250ms, for example, with high gain for highest SNR, and signals beingbinned together. To capture mid-range imagery photodetector “2”, locatedon the right 404 of the top row 406 of the pixel 400, can integrate for25 ms and be read out in low gain, for example. Furthermore, to capturethe high-range imagery pixel “3”, located on the left 402 of the bottomrow 408 of the pixel 400, can integrate for 1.25 ms and be read out inlow gain, for example. As described, gain may be controlled byselectively coupling a gain capacitor. In this manner, a 10,000,000:1dynamic range may be obtained, for example, assuming a noise floor of 1h+, a pinned photodiode saturation level of 50,000 h+ and the factor of200× in integration time.

The three signal outputs can then be measured and the best signal or acombination of the signals may then be scaled by integration time andused for display and radiometric accuracy analysis. In some aspects,non-linearity and calibration of the signal chain for each pixel can betaken into consideration. One of ordinary skill would readily recognizethat variations on the above mode of operation are also possible. Forinstance, long integration times can be greater than 250 ms and theshortest integration times can be the line time, which is typically lessthan 0.024 ms. Also, each photodetector may use a different integrationtime.

In the above-described mode of operation, handling the extremely widedynamic range may tax the anti-blooming capability in the quad-pixelconfiguration. To address this concern, the shortest integration timesignal may be read out first, followed by the middle integration time,and then the longest integration time. Furthermore, the transfer gatesmay be operated such that the pinned photodiodes can spill through thesense node to the reset drain before charge has a chance to spread toadjacent pixels.

Referring now to FIG. 5, an example timing diagram 550 for a controlsystem operating a pixel, in accordance with aspects of the presentdisclosure. Specifically, this example shows the operation of a pixelhaving two photodiodes in a row, both coupled to a sense node and havinga separate charge transfer gate coupled thereto. The pixel also includesa gain capacitor selectively connectable to the sense node, as well asreadout circuitry for measuring charge being accumulated in thephotodiodes. In some implementations, the readout circuitry includes areset transistor, a source follower transistor, a row select transistor,and a correlated double sampling (CDS) circuit. In particular, the CDScircuit may be located in proximity to the pixel or attached via acommon signal line to a location outside of the imaging area of thepixel.

Referring specifically to FIG. 5, the timing diagram 550 shows theactivation of a ROW SELECT, which chooses the row connected to a columnsignal line, a RESET, which resets the sense node diode voltage to areference voltage, and a MIM ENABLE, which connects or disconnects again capacitor from the sense node. The timing diagram 550 also showsactivation of a TG1, which is the charge transfer gate for the firstphotodiode, a TG2, which is the charge transfer gate for the secondphotodiode, a CLAMP, which activates the correlated double samplingcircuit (CDS) to store the value of sense node zero signal voltage, anda SAMPLE, which activates the CDS circuit to store the signal voltageafter charge is transferred to the sense node.

In operating the first photodiode, the exposure time is started byresetting the first photodiode charge to zero using the RESET.Specifically, the first integration time is started during the timeperiod labeled START EXPOSURE TIME, q. During the readout time of thefirst photodiode, the MIM ENABLE is shown to be off, and as such thecharge-to-voltage conversion is determined by the capacitance of thesense node. Although low noise is maintained, signal output saturationoccurs at a relatively low signal level. Similarly, in operating thesecond photodiode, the exposure time is started by resetting the secondphotodiode charge to zero using the RESET. Specifically, the secondintegration time is started during the time period labeled STARTEXPOSURE TIME, s. During the readout time of the second photodiode, theMIM ENABLE is shown to be on. This connects the gain capacitor to thesense node, thus modifying the capacitance across the sense node.Connecting the gain capacitor changes the charge-to-voltage conversionvalue and enables a higher charge signal to be stored on the sense node.

In this example, the row time was selected to be approximately 24 μs topermit both photodetectors to be read out with a ADC conversion rateconsistent with moderate rate column parallel ADCs. The clamp to sampletime of 7.5 μs was selected to be consistent with dominant time constantbeing targeted for low noise pixels and analog signal processing. Therow address n+k time slots was used to reset rows not being read out, toachieve sub-frame integration time. It may be readily appreciated thatother timings may be possible.

As shown in FIG. 5, the integration time of the first photodiode wasselected to be longer compared to the integration time of the secondphotodiode. This allows for an increased performance of the firstphotodiode at low light levels due to the longer exposure. On the otherhand, a higher number of photons can illuminate the second photodiodebefore saturation, at the cost of degraded low light performance due toincreased noise floor. However, by virtue of combining these into asingle pixel, a wide range of light signals can be measured. That is, byusing combining information from each of the photodiodes, a greaterintra-scene dynamic range can achieved, spanning low and high lightlevels.

As described with reference to FIGS. 1-4, pixels in accordance with thepresent disclosure may include more than two photodiodes in variousconfigurations, and signals produced therein may be readout outindividually or in various combinations. As such, the example timingdiagram 550 may be readily adapted to these variations. For example, ina quad-pixel configuration having four photodiodes, a different row ofcan be read out in a similar manner as described above, either using thesame or a different combination of timing and gain operations.

The present invention has been described in terms of one or moreembodiments, and it should be appreciated that many equivalents,alternatives, variations, and modifications, aside from those expresslystated, are possible and within the scope of the invention.

Additional Examples

Illustrative examples of the technologies disclosed herein are providedbelow. An embodiment of the technologies may include any one or more,and any combination of, the examples described below.

In an example 1, a system for imaging a scene described by light signalsthat extend over a wide dynamic range is provided. The system includesan imaging sensor comprising a plurality of pixels, each pixelcomprising a substrate, two or more photodiodes formed in the substratethat are coupled to a sense node, each photodiode having connectedthereto a different transfer gate for selectively controlling a chargeaccumulation therein due to an incident light, a gain capacitorselectively connectable to the sense node, and readout circuitry formeasuring the charge accumulation, the readout circuitry being connectedto the sense node and comprising a reset transistor, a source followertransistor, and a row select transistor. The system also includes acontroller configured to operate the imaging sensor by selecting adifferent integration time for at least two photodiodes in a pixel basedon light signals received by the pixel, performing a readout of thepixel, wherein a capacitance associated with the sense node is modifiedduring the readout of the at least two photodiodes, and assembling animage using signals obtained from the plurality of pixels in thereadout.

An example 2 includes the subject matter of example 1, wherein the atleast two photodiodes comprise pinned photodiodes.

An example 3 includes the subject matter of any of examples 1 and 2,wherein the at least two photodiodes comprise p-type diodes.

An example 4 includes the subject matter of any of examples 1, 2, and 3,wherein the gain capacitor is a metal insulator metal (“MIM”) capacitor.

An example 5 includes the subject matter of any of examples 1, 2, 3, and4, wherein each pixel comprises four photodiodes coupled to the sensenode.

An example 6 includes the subject matter of any of examples 1, 2, 3, 4,and 5, wherein a capacitance of the gain capacitor is selected basedupon a predetermined dynamic range of the imaging sensor.

An example 7 includes the subject matter of any of examples 1, 2, 3, 4,5, and 6, wherein the capacitance is in a range approximately between 1fF/μm² and 10 fF/μm².

An example 8 includes the subject matter of any of examples 1, 2, 3, 4,5, 6, and 7, wherein a size of each pixel is in a range approximatelybetween 1 and 20 μm.

An example 9 includes the subject matter of any of examples 1, 2, 3, 4,5, 6, 7, and 8, wherein at least one pixel in the array is responsive toa sub-band of a frequency spectrum describing the incident light.

An example 10 includes the subject matter of any of examples 1, 2, 3, 4,5, 6, 7, 8, and 9, wherein at least one pixel in the array comprises acolor filter.

In an example 11, a method for operating a pixel having two or morephotodiodes coupled to a sense node, each photodiode controlled using adifferent transfer gate, the methods includes the steps of positioningthe pixel to image a scene described by light signals that extend over awide dynamic range, and selecting a different integration time for atleast two photodiodes in the pixel based on light signals received bythe pixel. The method also includes the steps of performing a readout ofthe pixel using a readout circuit connected to the sense node, wherein acapacitance associated with the sense node is modified during thereadout of the at least two photodiodes.

An example 12 includes the subject matter of example 11, wherein thepixel comprises four pinned photodiodes coupled to the sense node.

An example 13 includes the subject matter of any of examples 11 and 12,wherein modifying the capacitance associated with the sense nodecomprises selectively connecting a gain capacitor to the sense node.

An example 14 includes the subject matter of any of examples 11, 12, and13, wherein the integration time is in a range approximately between 1ms and 1 sec.

An example 15 includes the subject matter of any of examples 11, 12, 13,and 14, wherein the method further comprises combining an accumulatedcharge of two or more photodiodes of the pixel.

An example 16 includes the subject matter of any of examples 11, 12, 13,14, and 15, wherein different integration times are selected for two ormore photodiodes to achieve a wide dynamic range for the pixel.

An example 17 includes the subject matter of any of examples 11, 12, 13,14, 15, and 16, wherein the wide dynamic range is up to approximately10,000,000:1.

An example 18 includes the subject matter of any of examples 11, 12, 13,14, 15, 16, and 17, wherein the method further comprises filtering theincident light received by one or more photodiodes in the pixel.

An example 19 includes the subject matter of any of examples 11, 12, 13,14, 15, 16, 17, and 18, wherein the method further comprises performinga readout of multiple pixels in an array to generate an image frame.

An example 20 includes the subject matter of any of examples 11, 12, 13,14, 15, 16, 17, 18, and 19, wherein the integration time of one or moreof the at least two photodiodes is selected based on information from aprevious image frame.

The invention claimed is:
 1. An apparatus, comprising: an image sensorhaving an overall set of pixels making up the image sensor to capture animage, where two or more pixels in the overall set of pixels each havean architecture that includes multiple photodiodes forming an individualpixel, where a control system is configured to cooperate with eachphotodiode in the individual pixel to allow multiple pixels in theoverall set of pixels to operate in a different imaging-mode ofoperation simultaneously within a first row making up the image, where afirst set of pixels within the first row in a first window regionoperate in a first imaging-mode of operation while simultaneously asecond set of pixels within the first row in a second window regionoperate in a second imaging-mode of operation, and where the imagesensor has multiple window regions, including the first window regionand the second window region, making up a full field of vision of theimage captured by the image sensor.
 2. The apparatus of claim 1, wherethe image sensor is fabricated with at least two or more semiconductorlayers, with a first layer having at least the two or more pixels havingthe architecture that includes the multiple photodiodes, and a secondlayer utilized to route wire traces for control signals includingcontrol signals going to i) transfer gates for each of the multiplephotodiodes, and ii) a gate for a gain capacitor in each pixel of theimage sensor in order to enable the different imaging-modes of operationfor each pixel in the first row on a per pixel basis.
 3. The apparatusof claim 1, where the control system is configured to use one or moredecoders cooperating with a timer to direct control signals to i)transfer gates for each of the multiple photodiodes in the individualpixel to control an integration time for that photodiode, ii) a gate fora gain capacitor connecting to a common sense node to alter a gainapplied to an accumulated charge from one or more of the photodiodesbeing sensed and read out, and iii) a gate for row select, where a firstpixel is directed to operate in the first imaging-mode of operation inthe first window and a second pixel in the first row is directed via thecontrol signals to operate in the second imaging-mode of operationwithin the second window of the image.
 4. The apparatus of claim 3,where the control system is configured to use a first decodercooperating with the timer to direct the control signals to the transfergates for each of the multiple photodiodes and the gate for the rowselect transistor to tell the image sensor what one or more pixellocations and what row of the image sensor will be readout, and a seconddecoder cooperating with the timer to direct the control signals tocontrol an integration time for each photodiode and whether the gaincapacitor should connect or not connect to the common sense node tocontrol gain.
 5. The apparatus of claim 1, where the control system isconfigured to use a controller cooperating with a timer to send controlsignals to bin together any combination of two or more photodiodes tocollect and read out accumulated charge, at a same time, from thephotodiodes binned together, where the controller cooperating with thetimer is configured to pulse gates of all of the photodiodes binnedtogether at a same time with other binned photodiodes of the controlsystem while a remaining amount of photodiodes in the individual pixelhave their control signals sent at a different time from the photodiodesbinned together.
 6. The apparatus of claim 1, where the control systemis configured to use a controller and a timer to cooperate to enable anextended dynamic range imaging-mode of operation via i) having one ormore pixels in a row configured to operate in a binned imaging-mode ofoperation to capture charge from the first set of pixels in the firstrow that are in low light conditions in the first window while ii)having one or more pixels in the second set of pixels in the first rowconfigured to operate with a short integration time and low gainimaging-mode of operation to capture charge from the pixels in thesecond set of pixels in the image that are in bright light conditions inthe second window, as well as iii) having other pixels in the first rowconfigured to operate in a high frame rate imaging-mode of operation toeliminate blur in the other pixels when recording moving objects in athird window.
 7. The apparatus of claim 1, where the control system isconfigured to use a controller and a timer cooperating to send controlsignals to enable a first pixel in the first row in the first window tooperate in an extended dynamic range imaging-mode of operation while asecond pixel in the first row to operate in a high frame rateimaging-mode of operation in the second window based on sensed imagingconditions in the first pixel and the second pixel.
 8. The apparatus ofclaim 1, where the control system is configured to use two or moredecoders cooperating with a timer to direct control signals to controli) frame rate, ii) integration times of the multiple photodiodes, andiii) binning of the multiple photodiodes on a per pixel basis to allowmultiple pixels in the overall set of pixels to operate in a sameimaging-mode of operation simultaneously within each row within a thirdwindow, all within a time frame of one image frame.
 9. The apparatus ofclaim 1, where the control system is configured to use a controller anda timer that cooperate to place a first pixel in the first row into anextended dynamic-range imaging-mode of operation in the first windowwhile sending control signals to a second pixel in the second window tooperate in the second imaging-mode of operation, where the controllerand the timer that cooperate send control signals to the multiplephotodiodes such that at least two of the photodiodes are binnedtogether.
 10. The apparatus of claim 1, where the control system isconfigured to use one or more decoders cooperating with a timer and wiretraces routed in a metallization layer, where the one or more decoderscooperating with the timer and the wire traces routed in themetallization layer direct control signals to enable each pixel tooperate independently of another pixel, on a per pixel basis, in thefirst row.
 11. A method for an image sensor, comprising: fabricating animage sensor having an overall set of pixels making up the image sensorto capture an image, fabricating two or more pixels in the overall setof pixels to each have an architecture that includes multiplephotodiodes forming an individual pixel, and configuring a controlsystem to cooperate with each photodiode in the individual pixel toallow multiple pixels in the overall set of pixels to operate in adifferent imaging-mode of operation simultaneously within a first rowmaking up the image, where the image sensor has multiple window regionsmaking up a full field of vision of the image captured by the imagesensor, where a first set of pixels within the first row in a firstwindow region operate in a first imaging-mode of operation whilesimultaneously a second set of pixels within the first row in a secondwindow region operate in a second imaging-mode of operation.
 12. Themethod of claim 11, further comprising: fabricating the image sensorwith at least two or more semiconductor layers, with a first layerhaving pixels, and a second layer utilized to generate control signalsand route wire traces for control signals including control signalsgoing to i) transfer gates for each of the multiple photodiodes, and ii)a gate for a gain capacitor in each pixel in the first layer on afabricated chip in order to enable the first and second imaging-modes ofoperation for each pixel in the first row on a per pixel; and thus, themultiple pixels in the overall set of pixels to operate in the differentimaging modes of operation mode simultaneously within the first row inthe first window and the second window.
 13. The method of claim 11,further comprising: configuring the control system to use one or moredecoders cooperating with a timer to direct control signals to i)transfer gates for each of the multiple photodiodes in the individualpixel to control an integration time for that photodiode, ii) a gate fora gain capacitor connecting to a common sense node to alter a gainapplied to an accumulated charge from one or more of the photodiodesbeing sensed and read out, and iii) a gate for row select transistor,where a first pixel is directed to operate in the first imaging-mode ofoperation and a second pixel in the first row is directed via thecontrol signals to operate in the second imaging-mode of operation. 14.The method of claim 13, further comprising: configuring the controlsystem to use a first decoder cooperating with the timer to direct thecontrol signals to i) the transfer gates for each of the multiplephotodiodes and iii) the gate for the row select transistor to tell theimage sensor what one or more pixel locations and what row of the imagesensor will be readout, and a second decoder cooperating with the timerto direct the control signals to control an integration time for eachphotodiode and whether the gain capacitor should connect or not connectto the common sense node to control gain.
 15. The method of claim 11,further comprising: configuring the control system to use a controllercooperating with a timer to send control signals to bin together anycombination of two or more photodiodes to collect and read outaccumulated charge from the photodiodes binned together, where all ofthe photodiodes binned together will have their transfer gates pulsed ata same time with other binned photodiodes of the control system while aremaining amount of photodiodes in the individual pixel will have theircontrol pulses occur at a different time from the photodiodes binnedtogether, where photodiodes diagonally aligned in the individual pixelare binned together first.
 16. The method of claim 11, furthercomprising: configuring the control system to use two or more decoderscooperating with a timer to direct control signals to control i) framerate, ii) integration times of the multiple photodiodes, and iii)binning of the multiple photodiodes on a per pixel basis to allowmultiple pixels in the overall set of pixels to operate in a sameimaging-mode of operation simultaneously within each row within a thirdwindow, all within a time frame of one image frame.
 17. The method ofclaim 11, further comprising: configuring the control system to acontroller and a timer to cooperate to send control signals to enable afirst pixel the overall set of pixels in the first window to operate inan extended dynamic-range imaging-mode of operation while a second pixelthe overall set of pixels in the second window operates in a high framerate imaging-mode of operation based on sensed imaging conditions in thefirst pixel and imaging conditions in the second pixel.
 18. The methodof claim 11, further comprising: configuring the control system to useone or more decoders cooperating with a timer to direct control signalsto one or more rows in the first window to operate in a high frame rateimaging-mode of operation and to a first pixel in the first row tooperate with two or more photodiodes binned together with a second pixelin the first row based on sensed imaging conditions in the first pixeland imaging conditions in the second pixel.
 19. The method of claim 11,further comprising: configuring the control system to use a controllerand a timer to cooperate to enable having one or more pixels in thefirst row in the first window to operate in a binned imaging-mode ofoperation to capture charge from the first set of in the first row thatare in low light conditions while having one or more pixels in the firstrow in the second window to operate in a short integration time and lowgain imaging-mode of operation to capture charge from the second set ofpixels in the image that are in bright light conditions, and havingother pixels in the first row when recording moving objects to operatein a high frame rate imaging-mode of operation to eliminate blur in theother pixels in a third window.
 20. The method of claim 11, furthercomprising: configuring the control system to use one or more decoderscooperating with a timer and wire traces routed in a metallization,where the one or more decoders cooperating with the timer and the wiretraces routed in the metallization layer direct control signals toenable each pixel in the first row to operate independently of anotherpixel, on a per pixel basis.